Self-cleaning sensor system of a motor vehicle

ABSTRACT

A self-cleaning sensor system of a motor vehicle includes an object sensor having a lens surface facing a region located external to the motor vehicle. The object sensor generates a signal associated with an image or a video of the region. The system further includes multiple lens treatment devices for applying remedies for removing an obscurity formed on the lens surface. The system further includes a computer having one or more processors and a computer readable medium storing instructions. The processor is programmed to determine a classification of the obscurity, in response to the processor receiving the signal from the object sensor. The processor is further programmed to generate an actuation signal, and the associated lens treatment device applies the remedy, in response to the associated lens treatment device receiving the actuation signal from the processor.

INTRODUCTION

The present disclosure relates to a motor vehicle having one or moreobject sensors, and more particularly to a self-cleaning sensor systemof a motor vehicle including an object sensor with a lens surface and alens treatment device for removing an obscurity formed on the lenssurface.

Automotive systems can include one or more object sensors to assistdrivers with operating the vehicle or to fully operate the vehicle. Forinstance, an Advanced Driver Assistance System (ADAS) typically includesmultiple object sensors in the form of one or more cameras for capturingimages of associated regions disposed external to the vehicle. Thecamera may be used to determine the presence of objects relative to theposition of the vehicle. More specifically, the camera can be used todetermine the presence of buildings and trees along a roadway, the speedof the vehicle, the position of the vehicle on the roadway (i.e.,lane-keeping), and the positions of other vehicles, pedestrians, orother objects that may be moving closer or farther away from thevehicle.

The cameras can be exposed to the environment, and unpredictableinclement weather conditions can severely impair the performance of thecameras. In contrast to hardware malfunctions, impairment caused byinclement weather can be difficult to predict and detect. Images fromimpaired sensors can lead to false detection of downstream processingmodules. Objects that appear in the impaired region could be missed, anda vehicle system can incorrectly recognize the impaired region itself asa false object.

Thus, while existing methods and systems may achieve their intendedpurpose, there is a need for a method and self-cleaning sensor systemthat address these issues.

SUMMARY

According to several aspects of the present disclosure, a self-cleaningsensor system of a motor vehicle includes multiple object sensors, witheach object sensor having a lens surface facing an associated one ofmultiple regions located external to the motor vehicle. Each objectsensor generates a signal associated with an image and/or a video of theassociated region as captured by the object sensor. The system furtherincludes a multiple lens treatment devices for applying an associatedone of multiple remedies to the lens surface of each object sensor. Theremedies are different from one another, and each remedy is configuredto remove an associated classification of an obscurity formed on thelens surface of the associated object sensor. The system furtherincludes a computer including one or more processors electricallycommunicating with the object sensors and the lens treatment devices.The computer further includes a non-transitory computer readable storagemedium storing instructions. The processor is programmed to utilize amulti-task neural network to determine a classification of theobscurity, in response to the processor receiving the signal from theobject sensor. The processor is further programmed to generate anactuation signal based on the classification of the obscurity. Theprocessor is further programmed to transmit the actuation signal to theassociated lens treatment device. The associated lens treatment deviceapplies the associated remedy to the lens surface, in response to theassociated lens treatment device receiving the actuation signal from theprocessor.

In one aspect, the lens treatment devices include a heat-based lenstreatment device for applying heat to the lens surface to remove theobscurity formed on the lens surface of the associated object sensor.

In another aspect, the processor is further programmed to determine thatthe classification of the obscurity is associated with an ice depositformed on the lens surface, in response to the processor receiving thesignal from the object sensor.

In another aspect, the lens treatment devices include a liquid-basedlens treatment device for delivering a pressurized liquid to the lenssurface to remove the obscurity formed on the lens surface of theassociated object sensor.

In another aspect, the processor is further programmed to determine thatthe classification of the obscurity is associated with a dirt depositformed on the lens surface, in response to the processor receiving thesignal from the object sensor.

In another aspect, the lens treatment devices include a gas-based lenstreatment device for delivering a compressed gas to the lens surface toremove the obscurity formed on the lens surface of the associated objectsensor.

In another aspect, the processor is further programmed to determine thatthe classification of the obscurity is associated with a liquid depositformed on the lens surface, in response to the processor receiving thesignal from the object sensor.

In another aspect, the processor is further programmed to determine thatat least a portion of the obscurity is disposed on the lens surfaceafter the lens treatment device applied the associated remedy to thelens surface. The processor is further programmed to count a number ofattempts taken to remove the obscurity and compare the number ofattempts to a maximum threshold. The processor is further programmed togenerate the actuation signal, in response to the processor determiningthat the number of attempts is below the maximum threshold. Theassociated lens treatment device applies the associated remedy to thelens surface, in response to the associated lens treatment devicereceiving the actuation signal from the processor.

According to several aspects of the present disclosure, a self-cleaningsensor system of a motor vehicle includes multiple object sensors havinga lens surface facing an associated one of multiple regions locatedexternal to the motor vehicle. Each object sensor generates an objectsignal associated with an image and/or a video of the associated regionas captured by the object sensor. The system further includes multiplelens treatment devices for applying an associated one of multipleremedies to the lens surface of each object sensor. The remedies aredifferent from one another, and each remedy is configured to remove anassociated classification of an obscurity formed on the lens surface ofthe associated object sensor. The lens treatment devices include aprimary lens treatment device for applying a primary remedy to the lenssurface to remove a primary classification of the obscurity. The systemfurther includes a computer having one or more processors electricallycommunicating with the object sensors and the lens treatment devices.The computer further includes a non-transitory computer readable storagemedium storing instructions. The processor is programmed to utilize amulti-task neural network to determine a primary classification of theobscurity, in response to the processor receiving the object signal fromthe object sensor. The processor is programmed to generate a primaryactuation signal based on the primary classification of the obscurity.The processor is programmed to transmit the primary actuation signal tothe primary lens treatment device. The primary lens treatment deviceapplies the primary remedy to the lens surface, in response to theprimary lens treatment device receiving the primary actuation signalfrom the processor.

In one aspect, the processor is further programmed to determine that atleast a portion of the obscurity is still disposed on the lens surfaceafter the primary lens treatment device applied the primary remedy tothe lens surface. The processor is further programmed to count a numberof attempts taken to remove the obscurity and compare the number ofattempts to a maximum threshold. The processor is further programmed todetermine a secondary classification of the obscurity, in response tothe processor determining that the number of attempts is below themaximum threshold and the processor further determining that the primaryremedy did not remove the entire obscurity. The processor is furtherprogrammed to generate a secondary actuation signal based on thesecondary classification. The lens treatment devices further include asecondary lens treatment device for applying a secondary remedy to thelens surface, in response to the secondary lens treatment devicereceiving the secondary actuation signal from the processor.

In another aspect, the processor is further programmed to determine thatat least a portion of the obscurity is still disposed on the lenssurface after the secondary lens treatment device applied the secondremedy to the lens surface. The processor is further programmed to countthe number of attempts taken to remove the obscurity and compare thenumber of attempts to the maximum threshold. The processor is furtherprogrammed to determine a tertiary classification of the obscurity, inresponse to the processor determining that the number of attempts isbelow the maximum threshold and the processor further determining thatthe secondary remedy did not remove the entire obscurity. The processoris further programmed to generate a tertiary actuation signal based onthe tertiary classification. The lens treatment devices further includea tertiary lens treatment device for applying a tertiary remedy to thelens surface, in response to the tertiary lens treatment devicereceiving the tertiary actuation signal from the processor.

In another aspect, the system further includes one or more supplementalsensors for generating a supplemental signal associated with theobscurity, with the processor determining an accuracy of the primaryclassification, the secondary classification, and the tertiaryclassification based on the supplemental signal.

In another aspect, the supplemental sensors include a temperature sensorfor generating a temperature signal indicative of an ambienttemperature. One of the primary classification, the secondaryclassification, and the tertiary classification of the obscurity isassociated with an ice deposit formed on the lens surface. The processordetermines that the classification is not accurate, in response to thetemperature signal indicating that the ambient temperature is above afreezing temperature.

In another aspect, the primary lens treatment device, the secondary lenstreatment device, and the tertiary lens treatment device are anassociated one of a gas-based lens treatment device for delivering acompressed gas to the lens surface, a liquid-based lens treatment devicefor delivering a pressurized liquid to the lens surface, and aheat-based lens treatment device for applying heat to the lens surfaceto remove the obscurity formed on the lens surface of the associatedobject sensor.

According to several aspects of the present disclosure, a method isprovided for operating a self-cleaning sensor system for a motorvehicle. The system includes multiple object sensors, with each objectsensor having a lens surface facing an associated one of multipleregions located external to the motor vehicle. The system furtherincludes multiple lens treatment devices including a primary lenstreatment device, a secondary lens treatment device, and a tertiary lenstreatment device. The system further includes one or more supplementalsensors and a computer having one or more processors and anon-transitory computer readable storage medium storing instructions.The method includes generating, using the object sensor, an objectsignal associated with an image and/or a video of the region captured bythe object sensor. The method further includes determining, using theprocessor, a primary classification of the obscurity in response to theprocessor receiving the object signal from the object sensor. The methodfurther includes generating, using the processor, a primary actuationsignal based on the primary classification of the obscurity. The methodfurther includes transmitting, using the processor, the primaryactuation signal to the primary lens treatment device. The methodfurther includes applying, using the primary lens treatment device, aprimary remedy to the lens surface in response to the primary lenstreatment device receiving the primary actuation signal from theprocessor.

In one aspect, the method further includes determining, using theprocessor, whether the obscurity is still disposed on the lens surfaceafter the primary lens treatment device applied the primary remedy tothe lens surface. The method further includes counting, using theprocessor, a number of attempts taken to remove the obscurity andcomparing, using the processor, the number of attempts to a maximumthreshold. The method further includes determining, using the processor,a secondary classification of the obscurity in response to the processordetermining that the number of attempts is below the maximum thresholdand the processor further determining that the primary remedy did notremove the entire obscurity. The method further includes generating,using the processor, a secondary actuation signal based on the secondaryclassification. The method further includes applying, using thesecondary lens treatment device, a secondary remedy to the lens surfacein response to the secondary lens treatment device receiving thesecondary actuation signal from the processor.

In another aspect, the method further includes determining, using theprocessor, whether the obscurity is still disposed on the lens surfaceafter the secondary lens treatment device applied the secondary remedyto the lens surface. The method further includes counting, using theprocessor, the number of attempts taken to remove the obscurity andcomparing, using the processor, the number of attempts to the maximumthreshold. The method further includes determining, using the processor,a tertiary classification of the obscurity in response to the processordetermining that the number of attempts is below the maximum thresholdand the processor further determining that the secondary remedy did notremove the obscurity. The method further includes generating, using theprocessor, a tertiary actuation signal based on the tertiaryclassification. The method further includes applying, using the tertiarylens treatment device, a tertiary remedy to the lens surface in responseto the tertiary lens treatment device receiving the tertiary actuationsignal from the processor.

In another aspect, the method further includes generating, using thesupplemental sensor, a supplemental signal associated with theobscurity. The method further includes determining, using the processor,an accuracy of the primary classification, the secondary classification,and/or the tertiary classification based on the supplemental signal.

In another aspect, the method further includes generating, using atemperature sensor, a temperature signal indicative of an ambienttemperature. The method further includes determining, using theprocessor, that the primary classification, the secondaryclassification, and/or the tertiary classification associated with anice deposit is not accurate in response to the temperature signalindicating that the ambient temperature is above a freezing temperature.

In another aspect, the method further includes delivering, using agas-based lens treatment device, a compressed gas to the lens surface toremove the obscurity formed on the lens surface of the object sensor.The method further includes delivering, using a liquid-based lenstreatment device, a pressurized liquid to the lens surface to remove theobscurity formed on the lens surface of the object sensor. The methodfurther includes applying, using a heat-based lens treatment device,heat to the lens surface to remove the obscurity formed on the lenssurface of the object sensor.

Further areas of applicability will become apparent from the descriptionprovided herein. It should be understood that the description andspecific examples are intended for purposes of illustration only and arenot intended to limit the scope of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings described herein are for illustration purposes only and arenot intended to limit the scope of the present disclosure in any way.

FIG. 1 is a schematic diagram of one example of a motor vehicle having aself-cleaning sensor system including one or more object sensors fordetecting multiple objects positioned within regions external to themotor vehicle and multiple lens treatment devices for removingobscurities formed on the object sensors.

FIG. 2 is an enlarged view of a portion of the self-cleaning sensorsystem of FIG. 1 , illustrating one of the object sensors with anobscurity formed on a lens surface and multiple supplemental sensors forcorroborating detection of the obscurity.

FIG. 3 is an enlarged view of another example of a self-cleaning sensorsystem of FIG. 2 , illustrating the system without the supplementalsensors.

FIG. 4 is a flow chart of one example of a method for operating theself-cleaning sensor system of FIG. 1 .

FIG. 5 is a flow chart of one example of a method for operating theself-cleaning sensor system of FIG. 3 .

DETAILED DESCRIPTION

The following description is merely exemplary in nature and is notintended to limit the present disclosure, application, or uses.

Referring to FIG. 1 , one non-limiting example of a motor vehicle 100includes a self-cleaning sensor system 102 having multiple objectsensors 104 and multiple lens treatment devices 106 for applyingassociated remedies to each object sensor 104 to remove an obscurity 128(FIG. 2 ) impairing the object sensor 104. The system 102 furtherincludes a computer 108 with a multi-task neural network configured tomonitor in real-time an image stream generated by the object sensor.Once the computer 108 determines that the object sensor 104 is impaired,the computer 108 outputs the classification of the obscurity (e.g.,frost, water drop, mud, etc.) and an optional 2D mask to indicateimpaired regions, which in turn triggers appropriate self-restoringprocedures.

Non-limiting examples of the motor vehicle can include a land vehicle,such as a sedan, a light duty truck, a heavy duty truck, a sport utilityvehicle, a van, or a motor home. The motor vehicle 100 can be anautonomous vehicle, a semi-autonomous vehicle, or a non-autonomous(manual) vehicle. Non-limiting examples of the object 110 can include atrain 112 a, a streetcar 112 b, another vehicle 112 c, a bicyclist 112d, a motorcyclist 112 e, an animal 112 f, or a pedestrian 112 g.However, it is contemplated that the system can detect other suitablemotor vehicles and objects that are positioned external to the motorvehicle and facing the object sensor 104.

Each of the object sensors 104 has a lens surface 114 (FIG. 2 ) oremitter surface that faces an associated one of the regions 116 a-116 dlocated external to the motor vehicle 100. The object sensor 104 isconfigured to generate a signal associated with an image and/or a videoof the associated region. The object sensors 104 can include a forwardfacing camera 118 a and a forward radar device 119 a attached to a frontend structure 120 a of the motor vehicle 100 for capturing images andvideo of objects positioned within the region 116 a that is locatedforward of the motor vehicle 100. The object sensors 104 can furtherinclude a rear facing camera 118 b and a rear facing radar device 119 battached to a rear end structure 120 b of the motor vehicle 100 forcapturing images and video of objects positioned within the region 116 bthat is located rearward of the motor vehicle 100. The object sensors104 can further include a driver side camera 118 c attached to a driverside 120 c of the motor vehicle 100 for capturing images and video ofobjects positioned within the region 116 c that is located proximal tothe driver side 120 c of the motor vehicle 100. The object sensors 104can further include a passenger side camera 118 d attached to apassenger side 120 d of the motor vehicle 100 for capturing images andvideo of objects positioned within the region 116 d that is locatedproximal to the passenger side 120 d of the motor vehicle 100. Thecameras can be monocular, stereo, or infrared cameras. Othernon-limiting examples of the object sensors 104 can include othercameras, radar devices, ultrasonic devices, and/or lidar devices mountedto any suitable portion of the motor vehicle 100. In still othernon-limiting examples, it is contemplated that the object sensors caninclude cameras or any other object sensor, which face an interior of apassenger cabin and may be susceptible to a cracked lens and/orcontaminants (e.g., dust, spilled beverages, spilled food, etc.).

Referring to FIG. 2 , each of the lens treatment devices 106 isconfigured to apply an associated one of multiple remedies to the lenssurface 114 to remove an associated classification of obscurity 128formed on the lens surface 114 of the object sensor 104. The remediescan be different from one another, and each one of the remedies isconfigured to remove an associated classification of obscurity 128formed on the lens surface 114. More specifically, one non-limitingexample of the lens treatment devices 106 can be a gas-based lenstreatment device 122 a for delivering a compressed gas to the lenssurface 114 to remove the obscurity 128 (e.g., water droplets) formed onthe lens surface 114. Another non-limiting example of the lens treatmentdevices 106 can be a liquid-based lens treatment device 122 b fordelivering a pressurized liquid to the lens surface 114 to remove theobscurity 128 (e.g., mud, road dirt, etc.) formed on the lens surface114. Still another non-limiting example of the lens treatment devices106 can be a heat-based lens treatment device 122 c (e.g., heatresistive wire element) for applying heat (e.g., radiative heat) to thelens surface 114 to remove the obscurity 128 (e.g., ice or a layer ofice) on the lens surface 114. In other examples, the system 102 caninclude any combination of these or other suitable lens treatmentdevices for removing obscurities from the object sensors 104.

The system 102 further includes the computer 108 having one or moreprocessors 124 electrically communicating with the object sensors 104and the lens treatment devices 106. The computer 108 further includes anon-transitory computer readable storage medium 126 storinginstructions. The processor 124 is programmed to utilize a classifier,such as a multi-task deep neural network (“DNN”), to determine aclassification or category of the obscurity 128 and the regionassociated with the object sensor having the obscurity 128, in responseto the processor 124 receiving the signals from the object sensors 104.It is contemplated that the system can utilize other suitableclassifiers. In this non-limiting example, the DNN utilizes a “softmax”function to determine the classification by modeling the probabilitiesof the obscurity 128 belonging to each classification across allclassifications, which can be defined by Equation 1:

$\begin{matrix}{{{softmax}{}(x)} = {\frac{1}{{\sum}_{j = 1}^{K}\exp\left( x_{j} \right)}\begin{bmatrix}{\exp\left( x_{1} \right)} \\{\exp\left( x_{2} \right)} \\ \vdots \\{\exp\left( x_{K} \right)}\end{bmatrix}}} & {{Eq}.1}\end{matrix}$

where K represents a number of the classifications, x_(j) represents thejth element of the real value vector x from the DNN, and softmax(x)represents a transformed real value vector, where all elements rangefrom (0,1) and sum up to 1.

Also in this non-limiting example, the DNN uses a binary mask toindicate the location and shape of the region where the impairment is inthe field of view of the associated object sensor 104. The DNN uses a“sigmoid” function to model a binary probability for each pixel todetermine that the pixel belongs to an impaired region or an intactregion, which can be defined by Equation 2:

$\begin{matrix}{{\sigma(x)} = \frac{1}{1 + e^{- x}}} & {{Eq}.2}\end{matrix}$

where x represents a real value from the DNN for each cell of a 2D grid,the shape of which corresponds to the shape of the original input image,and σ(x) represents a transformed real value in the range of (0,1).

More specifically, in this non-limiting example, the processor 124 isprogrammed to utilize the DNN to determine that the classification ofthe obscurity 128 is associated with a liquid deposit (e.g., raindroplets), a dirt deposit (e.g., mud or road dirt), ice (e.g., frost ora layer of ice), a cracked lens surface, and/or a wiper moving in thefield of view of the object sensor, in response to the processor 124receiving the signal from the associated object sensor 104. Theprocessor 124 is further configured to generate an actuation signalbased on the classification of the obscurity 128 and transmit theactuation signal to the associated lens treatment device 106.

The associated lens treatment device 106 applies the remedy to the lenssurface, in response to the associated lens treatment device 106receiving the actuation signal from the processor 124. Continuing withthe previous example, the gas-based lens treatment device 122 a deliversthe compressed gas to the lens surface 114 to remove the liquid deposit(e.g., rain drops) from the lens surface 114, in response to thegas-based lens treatment device 106 receiving the actuation signal fromthe processor 124. The liquid-based lens treatment device 122 b deliversthe pressurized liquid (e.g., cleaning fluid) to the lens surface 114 toremove the dirt deposit (e.g., mud or road dirt) from the lens surface114, in response to the liquid-based lens treatment device 106 receivingthe actuation signal from the processor 124. The heat-based lenstreatment device 122 c (e.g., the heat resistive wire element) appliesheat to the lens surface 114 to remove ice (e.g., frost or layer of ice)formed on the lens surface 114, in response to the heat-based lenstreatment device 106 receiving the actuation signal from the processor124.

Also, in this non-limiting example, the system 102 applies an iterationof the remedies until the processor 124 accurately determines theclassification of the obscurity and/or one of the lens treatment devices106 (e.g., a primary lens treatment device 122 a, a secondary lenstreatment device 122 b, a tertiary lens treatment device 122 c, etc.)has removed the entire obscurity from the object sensor 104. Morespecifically, the processor 124 is programmed to determine a primaryclassification of the obscurity 128, in response to the processor 124receiving the object signal from the object sensor 104. The processor124 is further programmed to generate a primary actuation signal basedon the primary classification of the obscurity 128. The processor 124 isfurther programmed to transmit the primary actuation signal to a primarylens treatment device 130. The primary lens treatment device 130 appliesa primary remedy to the lens surface 114, in response to the primarylens treatment device 130 receiving the primary actuation signal fromthe processor 124. It is contemplated that the system can include othersuitable lens treatment devices for applying associated remedies to thelens surface.

The processor 124 is further programmed to determine that at least aportion of the obscurity is still disposed on the lens surface 114 afterthe primary lens treatment device 130 applied the primary remedy to thelens surface 114. The processor 124 is further programmed to count anumber of attempts taken to remove the obscurity 128 and compare thenumber of attempts to a maximum threshold. The processor 124 is furtherprogrammed to determine a secondary classification of the obscurity, inresponse to the processor 124 determining that the number of attempts isbelow the maximum threshold and in further response to the processor 124determining that the primary remedy did not remove the entire obscurity.The processor 124 is further programmed to generate a secondaryactuation signal and transmit the secondary actuation signal to thesecondary lens treatment device 130, in response to the processor 124determining that the number of attempts is below the maximum threshold.The secondary lens treatment device 132 applies a secondary remedy tothe lens surface 114, in response to the secondary lens treatment device132 receiving the secondary actuation signal from the processor 124.

The processor 124 is further programmed to determine that at least aportion of the obscurity 128 is disposed on the lens surface 114 afterthe secondary lens treatment device 132 applied the secondary remedy tothe lens surface 114. The processor 124 is further programmed to countthe number of attempts taken to remove the obscurity 128 and compare thenumber of attempts to the maximum threshold. The processor 124 isfurther programmed to determine a tertiary classification of theobscurity 128, in response to the processor 124 determining that thenumber of attempts is below the maximum threshold and in furtherresponse to the processor 124 determining that the secondary remedy didnot remove the entire obscurity 128. The processor 124 is furtherprogrammed to generate a tertiary actuation signal based on the tertiaryclassification and transmit the tertiary actuation signal to thetertiary lens treatment device 134. The tertiary lens treatment device134 applies a tertiary remedy to the lens surface 114, in response tothe tertiary lens treatment device 134 receiving the tertiary actuationsignal from the processor 124.

The primary lens treatment device 130, the secondary lens treatmentdevice 132, and the tertiary lens treatment device 134 can be anyassociated one of the gas-based lens treatment device 122 a, theliquid-based lens treatment device 122 b, and the heat-based lenstreatment device 122 c.

Also, in this non-limiting example, the system 102 further includes oneor more supplemental sensors 136 for generating a supplemental signalassociated with the obscurity 128, where the processor 124 determines anaccuracy of the primary classification, the secondary classification,and the tertiary classification based on the supplemental signal. Inparticular, the supplemental sensors 136 include a temperature sensor138 generating a temperature signal indicative of an ambienttemperature. The processor 124 determines that the classification is notaccurate when the primary classification, the secondary classification,and the tertiary classification of the obscurity is associated with anice deposit (e.g., frost, a layer of ice, etc.) formed on the lenssurface 114, and in response to the temperature signal indicating thatthe ambient temperature is above a freezing temperature. It iscontemplated that other non-limiting examples of the supplemental sensorcan include a wheel encoder 139, a dirt detection sensor 140, or othersuitable sensors.

Referring to FIG. 3 , a self-cleaning system 202 is similar to thesystem 102 of FIG. 1 and has similar components identified by the samenumbers increased by 100. However, while the system 102 includessupplemental sensors 136 (e.g., the temperature sensor 138, a wheelencoder 139, and a dirt detection sensor 140) for multi-modalsensor-based triggering, the system 202 does not include supplementalsensors.

Referring to FIG. 4 , a method 300 for operating the self-cleaningsensor system of FIG. 1 begins at block 302, with the first and secondobject sensors of different modalities (e.g., the camera 118 a and theradar device 119 a directed to a common forward region 116 a or thecamera 118 b and the radar device 119 b directed to a common rear region116 b) generating associated first and second signals for images and/orvideos of objects positioned in the common region.

At block 304, the method 300 further includes determining, using theprocessor 124, whether the first and second object sensors 104corroborate one another. More specifically, if the processor 124determines that first and second object sensors 104 are only detectingcommon objects in the common region, the processor 124 determines thatneither one of the first and second object sensors 104 has an obscurityand the method 300 returns to block 302. If the processor 124 does notdetermine that the first and second object sensors are only detectingcommon objects in the common region (e.g., the processor determines thatone of the first and second object sensors 104 detects an object in thecommon region that is not detected by the other of the first and secondobject sensors 104), the processor 124 determines that the uncommonobject is an obscurity 128 and the method 300 proceeds to block 306 foreach one of the first and second object sensors 104. One non-limitingbenefit of determining consistency among the object sensors is that thesystem 102 can preserve computational and memory bandwidth, where theprocessor determines in block 302 that none of the first and secondobject sensors 104 has the obscurity. While blocks 302 through 340 aredirected to the first object sensor 104, the method 300 repeats blocks302 through 340 for the each one of the remaining object sensors 104. Inother examples, the method may not include blocks 302 and 304.

At block 306, the method 300 further includes determining, using the DNNof the processor 124, the primary classification of the obscurityaccording to Equation 1, in response to the processor 124 receiving thefirst object signal from the first object sensor 104. Non-limitingexamples of the primary classification can be associated with the liquiddeposit, the dirt deposit, the ice deposit, the cracked lens surface,and/or the wiper moving in the field of vision of the first objectsensor 104. In this non-limiting example, the primary classification canbe the liquid deposit.

At block 308, the method 300 further includes generating, using thesupplemental sensor 136, the supplemental signal associated with theobscurity and determining, using the processor 124, the accuracy of theprimary classification based on the supplemental signal. In onenon-limiting example, the primary classification is associated with theliquid deposit, and the supplemental sensor 136 is the temperaturesensor 138. More specifically, in this non-limiting example, the method300 includes generating, using the temperature sensor 138, thetemperature signal indicative of the ambient temperature. The method 300further includes determining, using the processor 124, that thetemperature sensor 138 does not corroborate the first object sensor 104and further that the primary classification (e.g., associated with theliquid deposit) is not accurate, in response to the processor 124determining that the ambient temperature is below a freezing temperaturebased on the temperature signal. In further response to same, the method300 proceeds to block 314. The method 300 further includes determining,using the processor 124, that the temperature sensor 138 corroboratesthe first object sensor 104 and further that the primary classificationassociated with the liquid deposit is accurate, in response to theprocessor 124 determining that the ambient temperature is above thefreezing temperature based on the temperature signal. In furtherresponse to same, the method 300 proceeds to block 310.

At block 310, the method 300 further includes generating, using theprocessor 124, the primary actuation signal based on the primaryclassification of the obscurity 128. The method 300 further includestransmitting, using the processor 124, the primary actuation signal tothe primary lens treatment device 130.

At block 312, the method 300 further includes applying, using theprimary lens treatment device 130, the primary remedy to the lenssurface 114 in response to the primary lens treatment device 130receiving the primary actuation signal from the processor 124.Continuing with the previous non-limiting example where the processor124 determines that the primary classification is associated with theliquid deposit, the primary lens treatment device 130 can be thegas-based lens treatment device 122 a (e.g., a compressed air deliverydevice) that applies the primary remedy in the form of the compressedgas to the lens surface 114 to remove the obscurity 128.

At block 314, the method 300 further includes counting, using theprocessor 124, a number of attempts taken to remove the obscurity 128and comparing, using the processor 124, the number of attempts to amaximum threshold. If the processor 124 determines that the number ofattempts is below the maximum threshold, the method 300 proceeds toblock 316. If the processor 124 determines that the number of attemptsis not below the maximum threshold, the method 300 proceeds to block340.

At block 316, the method 300 further includes determining, using theprocessor 124, whether any portion of the obscurity 128 still remains onthe lens surface 114 after the primary lens treatment device 130 appliedthe primary remedy to the lens surface 114. More specifically, if theprocessor 124 determines that any portion of the obscurity 128 remainson the lens surface 114 after the primary lens treatment device 130applied the primary remedy to the lens surface 114, the method 300proceeds to block 318. If the processor 124 does not determine that anyportion of the obscurity is still on the lens surface 114 after theprimary lens treatment device 130 applied the primary remedy to the lenssurface 114, the method 300 returns to block 302. This block can beperformed similar to block 304 where the system includes two or moresensors of different modalities directed to a common region.

At block 318, the method 300 further includes determining, using the DNNof the processor 124, the secondary classification of the obscuritybased on the first object signal, in response to the processor 124determining that the primary remedy did not remove the entire obscurity.Continuing with the previous example, the secondary classification isassociated with the dirt deposit.

At block 320, the method 300 further includes generating, using thesupplemental sensor 136, the supplemental signal associated with theobscurity and determining, using the processor 124, the accuracy of thesecondary classification based on the supplemental signal. In thisnon-limiting example, the secondary classification of the obscurity 128is associated with the dirt deposit, and the supplemental sensor 136 isa dirt detection sensor 140 (FIG. 2 ). The method 300 can includegenerating, using the dirt detection sensor 140, a dirt signalassociated with the dirt deposit. The method 300 further includesdetermining, using the processor 124, that the dirt detection sensor 140does not corroborate the first object sensor 104 and further that thesecondary classification (e.g., associated with the dirt deposit) is notaccurate, in response to the processor 124 determining that based on thedirt signal the dirt deposit is formed on the lens surface. In furtherresponse to same, the method 300 proceeds to block 326. The method 300further includes determining, using the processor 124, that the dirtdetection sensor 140 corroborates the first object sensor 104 andfurther that the secondary classification is accurate, in response tothe processor 124 determining that based on the dirt signal theobscurity (e.g., the dirt deposit) is formed on the lens surface 114. Infurther response to same, the method 300 proceeds to block 322.

At block 322, the method 300 further includes generating, using theprocessor 124, the secondary actuation signal based on the secondaryclassification of the obscurity. The method 300 further includestransmitting, using the processor 124, the secondary actuation signal tothe secondary lens treatment device 132.

At block 324, the method 300 further includes applying, using thesecondary lens treatment device 132, the secondary remedy to the lenssurface 114 in response to the secondary lens treatment device 132receiving the secondary actuation signal from the processor 124.Continuing with the previous example where the secondary classificationis associated with the dirt deposit, the secondary lens treatment device132 can be the liquid-based lens treatment device 122 b that applies thepressurized fluid delivered to the lens surface 114 to remove theobscurity 128.

At block 326, the method 300 further includes counting, using theprocessor 124, the number of attempts taken to remove the obscurity andcomparing, using the processor 124, the number of attempts to themaximum threshold. If the processor 124 determines that the number ofattempts is below the maximum threshold, the method 300 proceeds toblock 328. If the processor 124 determines that the number of attemptsis not below the maximum threshold, the method 300 proceeds to block340.

At block 328, the method 300 further includes determining, using theprocessor 124, whether any portion of the obscurity remains on the lenssurface 114 after the secondary lens treatment device 132 applied thesecondary remedy to the lens surface 114. If the processor 124determines that any portion of the obscurity remains on the lens surface114 after the secondary lens treatment device 132 applied the secondaryremedy to the lens surface 114, the method 300 proceeds to block 330. Ifthe processor 124 determines that no portion of the obscurity is on thelens surface 114 after the secondary lens treatment device 132 appliedthe secondary remedy to the lens surface 114, the method 300 returns toblock 302. This block can be performed similar to block 316.

At block 330, the method 300 further includes determining, using theprocessor 124, the tertiary classification of the obscurity based on thefirst object signal and in response to the processor 124 determiningthat the number of attempts is below the maximum threshold and infurther response to the processor 124 determining that the tertiaryremedy did not remove the entire obscurity. Continuing with the previousnon-limiting example, the secondary classification is associated withthe dirt deposit, and the tertiary classification is associated with theice deposit.

At block 332, the method 300 further includes generating, using thesupplemental sensor 136, the supplemental signal associated with theobscurity and determining, using the processor 124, the accuracy of thetertiary classification based on the supplemental signal. In onenon-limiting example, the tertiary classification is associated with theice deposit, and the supplemental sensor 136 is the temperature sensor138. More specifically, in this non-limiting example, the method 300includes generating, using the temperature sensor 138, the temperaturesignal indicative of the ambient temperature. The method 300 furtherincludes determining, using the processor 124, that the temperaturesensor 138 does not corroborate the first object sensor 104 and furtherthat the tertiary classification associated with the ice deposit is notaccurate, in response to the processor 124 determining that based on thetemperature signal the ambient temperature is above the freezing point.In further response to same, the method 300 proceeds to block 338. Themethod 300 further includes determining, using the processor 124, thatthe temperature sensor 138 corroborates the first object sensor 104 andfurther that the tertiary classification associated with the ice depositis accurate, in response to the processor 124 determining that based onthe temperature signal the ambient temperature is below the freezingpoint. In further response to same, the method 300 proceeds to block334.

At block 334, the method 300 further includes generating, using theprocessor 124, the tertiary actuation signal based on the tertiaryclassification of the obscurity. The method 300 further includestransmitting, using the processor 124, the tertiary actuation signal tothe tertiary lens treatment device 134.

At block 336, the method 300 further includes applying, using thetertiary lens treatment device 134, the tertiary remedy to the lenssurface 114 in response to the tertiary lens treatment device 134receiving the tertiary actuation signal from the processor 124.Continuing with the previous example where the tertiary classificationis associated with the ice deposit, the tertiary lens treatment device134 can be the heat-based lens treatment device 122 c (e.g., aresistance wire heating element) that transfers heat to the lens surface114 to remove the obscurity 128.

At block 338, the method 300 further includes counting, using theprocessor 124, the number of attempts taken to remove the obscurity andcomparing, using the processor 124, the number of attempts to themaximum threshold. If the processor 124 determines that the number ofattempts is below the maximum threshold, the method 300 returns to block302. If the processor 124 determines that the number of attempts is notbelow the maximum threshold, the method 300 proceeds to block 340. It iscontemplated that other examples of the method can have more or fewerthan three classifications and/or more or fewer than three lenstreatment devices that apply associated remedies different from oneanother.

At block 340, the method 300 further includes generating, using theprocessor 124, an error report signal in response to the processor 124determining that the number of attempts to restore the impaired objectsensors is not below the maximum threshold. The report signal isassociated with the impaired regions and the object sensors withobscurities to one or more downstream vehicle systems 142 (FIG. 2 ),such that those downstream vehicle systems 142 may adjust their affectedfunctions only based on the regions that are intact. For example, manyautomotive cameras are shipped with auto-exposure feature inside.Without impairment information, exposure parameters can only beoptimized globally, which can be misleading when portion of the imagecontent is invalid. By injecting impairment status and location to theimaging pipeline, exposure optimization can be based on the intactregion only, which results in optimal exposure control. By injectingimpairment status and location to the downstream pipeline, system canscreen out any results produced in the impaired regions and look toother intact sensors with overlapped field of view to cover the impairedregion.

Referring to FIG. 5 , a method 400 is similar to the method 300 of FIG.3 . However, while the method 300 includes blocks 308, 320, 332 directedto the supplemental sensors 136 corroborating one or more object sensors104 (e.g., the cameras) for multi-modal sensor-based triggering, themethod 400 does not include blocks 308, 320, 332 utilizing thesupplemental sensors to corroborate the object sensors.

By getting impairment status as well as the location of impaired region,the sensor may adjust its affected functions only based on the regionthat is intact. For example, many automotive cameras are shipped withauto-exposure feature inside. Without impairment information, exposureparameters can only be optimized globally, which can be misleading whenportion of the image content is invalid. By injecting impairment statusand location to the imaging pipeline, exposure optimization can be basedon the intact region only, which results in best exposure control.

The description of the present disclosure is merely exemplary in natureand variations that do not depart from the gist of the presentdisclosure are intended to be within the scope of the presentdisclosure. Such variations are not to be regarded as a departure fromthe spirit and scope of the present disclosure.

What is claimed is:
 1. A self-cleaning sensor system of a motor vehicle,the self-cleaning sensor system comprising: a plurality of objectsensors, with each of the object sensors having a lens surface facing anassociated one of a plurality of regions that are different from oneanother and located external to the motor vehicle, with each of theobject sensors generating a signal associated with at least one of animage and a video of the associated regions; a plurality of lenstreatment devices for applying an associated one of a plurality ofremedies to the lens surface to remove an obscurity formed on the lenssurface of the associated object sensor, where the remedies aredifferent from one another and each of the remedies is configured toremove an associated classification of the obscurity from the lenssurface; a computer including at least one processor electricallycommunicating with each of the object sensors and the lens treatmentdevices, and the computer further including a non-transitory computerreadable storage medium storing instructions, such that the at least oneprocessor is programmed to utilize a multi-task neural network to:determine a classification of the obscurity in response to the at leastone processor receiving the signal from the associated object sensor;generate an actuation signal based on the classification of theobscurity; and transmit the actuation signal to the associated lenstreatment device; wherein the associated lens treatment device appliesthe associated remedy to the lens surface in response to the associatedlens treatment device receiving the actuation signal from the at leastone processor.
 2. The self-cleaning sensor system of claim 1 wherein theplurality of lens treatment devices include a heat-based lens treatmentdevice for applying heat to the lens surface to remove the obscurityformed on the lens surface of the associated object sensor.
 3. Theself-cleaning sensor system of claim 2 wherein the at least oneprocessor is further programmed to determine that the classification ofthe obscurity is associated with an ice deposit formed on the lenssurface in response to the at least one processor receiving the signalfrom the associated object sensor.
 4. The self-cleaning sensor system ofclaim 2 wherein the plurality of lens treatment devices further includea liquid-based lens treatment device for delivering a pressurized liquidto the lens surface to remove the obscurity formed on the lens surfaceof the associated object sensor.
 5. The self-cleaning sensor system ofclaim 4 wherein the at least one processor is further programmed todetermine that the classification of the obscurity is associated with adirt deposit formed on the lens surface in response to the at least oneprocessor receiving the signal from the associated object sensor.
 6. Theself-cleaning sensor system of claim 4 wherein the plurality of lenstreatment devices further include a gas-based lens treatment device fordelivering a compressed gas to the lens surface to remove the obscurityformed on the lens surface of the associated object sensor.
 7. Theself-cleaning sensor system of claim 6 wherein the at least oneprocessor is further programmed to determine that the classification ofthe obscurity is associated with a liquid deposit formed on the lenssurface in response to the at least one processor receiving the signalfrom the associated object sensor.
 8. The self-cleaning sensor system ofclaim 6 wherein the at least one processor is further programmed to:determine that at least a portion of the obscurity is disposed on thelens surface after the lens treatment device applied the associatedremedy to the lens surface; count a number of attempts taken to removethe obscurity; compare the number of attempts to a maximum threshold;and generate the actuation signal in response to the at least oneprocessor determining that the number of attempts is below the maximumthreshold; wherein the associated lens treatment device applies theassociated remedy to the lens surface in response to the associated lenstreatment device receiving the actuation signal from the at least oneprocessor.
 9. A self-cleaning sensor system of a motor vehicle, theself-cleaning sensor system comprising: a plurality of object sensors,with each of the object sensors having a lens surface facing anassociated one of a plurality of regions that are different from oneanother and located external to the motor vehicle, with each of theobject sensors generating an object signal associated with at least oneof an image and a video of the associated regions; a plurality of lenstreatment devices for applying an associated one of a plurality ofremedies to the lens surface to remove an obscurity formed on the lenssurface of the associated object sensor, where the remedies aredifferent from one another and each of the remedies is configured toremove an associated classification of the obscurity from the lenssurface, and the lens treatment devices include a primary lens treatmentdevice for applying a primary remedy to the lens surface; a computerincluding at least one processor electrically communicating with each ofthe object sensors and the lens treatment devices, and the computerfurther including a non-transitory computer readable storage mediumstoring instructions, such that the at least one processor is programmedto utilize a multi-task neural network to: determine a primaryclassification of the obscurity in response to the at least oneprocessor receiving the object signal from the associated object sensor;generate a primary actuation signal based on the primary classificationof the obscurity; and transmit the primary actuation signal to theassociated lens treatment device; wherein the primary lens treatmentdevice applies the primary remedy to the lens surface in response to theprimary lens treatment device receiving the primary actuation signalfrom the at least one processor.
 10. The self-cleaning sensor system ofclaim 9 wherein the at least one processor is further programmed to:determine that at least a portion of the obscurity is disposed on thelens surface after the primary lens treatment device applied the primaryremedy to the lens surface; count a number of attempts taken to removethe obscurity; compare the number of attempts to a maximum threshold;and determine a secondary classification of the obscurity in response tothe at least one processor determining that the number of attempts isbelow the maximum threshold and the primary remedy did not remove theobscurity; generate a secondary actuation signal based on the secondaryclassification; and wherein the plurality of lens treatment devicesfurther include a secondary lens treatment device for applying asecondary remedy to the lens surface in response to the secondary lenstreatment device receiving the secondary actuation signal from the atleast one processor.
 11. The self-cleaning sensor system of claim 10wherein the at least one processor is further programmed to: determinethat at least a portion of the obscurity is disposed on the lens surfaceafter the secondary lens treatment device applied the secondary remedyto the lens surface; count the number of attempts taken to remove theobscurity; compare the number of attempts to the maximum threshold; anddetermine a tertiary classification of the obscurity in response to theat least one processor determining that the number of attempts is belowthe maximum threshold and the at least one processor further determiningthat the secondary remedy did not remove the obscurity; generate atertiary actuation signal based on the tertiary classification; andwherein the plurality of lens treatment devices further include atertiary lens treatment device for applying a tertiary remedy to thelens surface in response to the tertiary lens treatment device receivingthe tertiary actuation signal from the at least one processor.
 12. Theself-cleaning sensor system of claim 11 further comprising at least onesupplemental sensor generating a supplemental signal associated with theobscurity, wherein the at least one processor determines an accuracy ofat least one of the primary classification, the secondaryclassification, and the tertiary classification based on thesupplemental signal.
 13. The self-cleaning sensor system of claim 12wherein the at least one supplemental sensor comprises a temperaturesensor for generating a temperature signal indicative of an ambienttemperature, and one of the primary classification, the secondaryclassification, and the tertiary classification of the obscurity isassociated with an ice deposit formed on the lens surface, and the atleast one processor determines that the primary classification of theice deposit is not accurate in response to the temperature signalindicating that the ambient temperature is above a freezing temperature.14. The self-cleaning sensor system of claim 13 wherein the primary lenstreatment device, the secondary lens treatment device, and the tertiarylens treatment device comprises: a gas-based lens treatment device fordelivering a compressed gas to the lens surface to remove the obscurityformed on the lens surface of the associated object sensor, aliquid-based lens treatment device for delivering a pressurized liquidto the lens surface to remove the obscurity formed on the lens surfaceof the associated object sensor; and a heat-based lens treatment devicefor applying heat to the lens surface to remove the obscurity formed onthe lens surface of the associated object sensor.
 15. A method ofoperating a self-cleaning sensor system for a motor vehicle, theself-cleaning sensor system having a plurality of object sensors, witheach of the object sensors having a lens surface facing an associatedone of a plurality of regions located external to the motor vehicle, theself-cleaning sensor system further having a plurality of lens treatmentdevices including a primary lens treatment device, a secondary lenstreatment device, and a tertiary lens treatment device, theself-cleaning sensor system further having at least one supplementalsensor, the self-cleaning sensor system further having a computerincluding at least one processor electrically communicating with each ofthe object sensors and the lens treatment devices, and the computerfurther including a non-transitory computer readable storage mediumstoring instructions, the method comprising: generating, using each ofthe object sensors, an object signal associated with at least one of animage and a video of the associated region; determining, using the atleast one processor, a primary classification of the obscurity inresponse to the at least one processor receiving the object signal fromthe associated object sensor; generating, using the at least oneprocessor, a primary actuation signal based on the primaryclassification of the obscurity; transmitting, using the at least oneprocessor, the primary actuation signal to the primary lens treatmentdevice; and applying, using the primary lens treatment device, a primaryremedy to the lens surface in response to the primary lens treatmentdevice receiving the primary actuation signal from the at least oneprocessor.
 16. The method of claim 15 further comprising: determining,using the at least one processor, whether the obscurity is disposed onthe lens surface after the primary lens treatment device applied theprimary remedy to the lens surface; counting, using the at least oneprocessor, a number of attempts taken to remove the obscurity;comparing, using the at least one processor, the number of attempts to amaximum threshold; and determining, using the at least one processor, asecondary classification of the obscurity in response to the at leastone processor determining that the number of attempts is below themaximum threshold and the at least one processor further determiningthat the primary remedy did not remove the obscurity; generating, usingthe at least one processor, a secondary actuation signal based on thesecondary classification; and applying, using the secondary lenstreatment device, a secondary remedy to the lens surface in response tothe associated lens treatment device receiving the secondary actuationsignal from the at least one processor.
 17. The method of claim 16further comprising: determining, using the at least one processor,whether the obscurity is disposed on the lens surface after thesecondary lens treatment device applied the secondary remedy to the lenssurface; counting, using the at least one processor, the number ofattempts taken to remove the obscurity; comparing, using the at leastone processor, the number of attempts to the maximum threshold; anddetermining, using the at least one processor, a tertiary classificationof the obscurity in response to the at least one processor determiningthat the number of attempts is below the maximum threshold and the atleast one processor further determining that the secondary remedy didnot remove the obscurity; generating, using the at least one processor,a tertiary actuation signal based on the tertiary classification; andapplying, using the tertiary lens treatment device, a tertiary remedy tothe lens surface in response to the tertiary lens treatment devicereceiving the tertiary actuation signal from the at least one processor.18. The method of claim 17 further comprising: generating, using the atleast one supplemental sensor, a supplemental signal associated with theobscurity; and determining, using the at least one processor, anaccuracy of at least one of the primary classification, the secondaryclassification, and the tertiary classification based on thesupplemental signal.
 19. The method of claim 18 further comprising:generating, using a temperature sensor, a temperature signal indicativeof an ambient temperature; and determining, using the at least oneprocessor, that at least one of the primary classification, thesecondary classification, and the tertiary classification of theobscurity is associated with the ice deposit formed on the lens surfaceand is not accurate in response to the at least one processordetermining that the temperature signal indicates that the ambienttemperature is above a freezing temperature.
 20. The method of claim 19further comprising: delivering, using a gas-based lens treatment device,a compressed gas to the lens surface to remove the obscurity formed onthe lens surface of the object sensor; delivering, using a liquid-basedlens treatment device, a pressurized liquid to the lens surface toremove the obscurity formed on the lens surface of the object sensor;and applying, using a heat-based lens treatment device, heat to the lenssurface to remove the obscurity formed on the lens surface of the objectsensor.